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Test  and  Evaluation  for  the  “On-Demand”  World 

Steven  J.  Hutchison,  Ph.D. 

Test  and  Evaluation  Executive, 

Defense  Information  Systems  Agency,  Arlington,  Virginia 


efense  Information  Systems  Agency 
(DISA),  is  a  Combat  Support  Agen¬ 
cy  dedicated  to  providing  our  na¬ 
tion’s  leaders  and  war¬ 
fighters  the  best  infor¬ 
mation  technology  solutions  needed  to 
accomplish  the  mission.  We  provision, 
operate,  and  secure  the  Global  Informa¬ 
tion  Grid  (GIG)  in  support  of  net-centric 
operations.  DISA  is  also  an  acquisition 
organization  with  a  large  test  and  evalu¬ 
ation  (T8cE)  workforce  that  specializes  in 
T5cE  for  information  technologies  (IT). 

Our  vision  in  DISA  T8cE  is  to  be  a 
center  of  excellence  for  advancing  the  art 
and  science  of  T8cE  for  IT.  Toward  that 
vision,  we  have  established  four  goals: 

•  Goal  1:  Support  the  warfighter — provide  respon¬ 
sive,  relevant,  value-added  test  and  evaluation 
services. 

•  Goal  2:  Be  the  premier  tester  and  evaluator  of 
Joint  net-centric  warfighting  capabilities  in  the 
Department  of  Defense  (DoD). 

•  Goal  3:  Team  for  maximum  efficiency.  Empower 
our  teams  to  design  and  execute  robust  test  plans 
and  provide  objective  assessments  to  decision 
makers. 

•  Goal  4:  Develop  and  retain  a  highly  qualified 
workforce. 

Bottom  line — My  job  is  to  help  our  programs  field 
enhanced  IT  capabilities  to  our  warfighters.  I  do  that 
by  ensuring  that  our  T5cE  organizations  are  engaged 
with  the  programs  to  help  them  find  and  fix  problems 
early,  stay  objective  in  their  assessments,  and  invest  in 
the  right  technologies  that  will  keep  us  relevant  in  the 
fast-paced,  ever- changing,  “on-demand”  world  of  IT. 

In  DISA,  we  are  trying  to  shift  acquisition  and 
testing  of  IT  to  be  more  responsive  to  warfighters 
needs — our  first  goal.  The  shift  that  I’m  addressing  is 
highlighted  in  the  current  strategic  planning  guidance 
(SPG):  “Evolve  planning,  determination  of  needed 
capabilities,  acquisition,  test  and  evaluation,  program¬ 
ming  and  training  toward  ‘demand  driven’  processes 


focused  increasingly  on  developing  and  testing  joint 
capability  portfolios  to  meet  Combatant  Commander 
preferred  approaches.”  Demand-driven  processes.  Joint 
capability  portfolios,  and  combatant 
commander  approaches  aU  signal  change 
...and  remind  us  of  who  we  work 
for — the  warfighter. 

We  are  expending  a  lot  of  energy  and 
intellectual  capital  on  finding  ways  to 
improve  our  acquisition  and  T5cE  pro¬ 
cesses.  Many  of  you  are  aware  that 
DISA  is  the  sponsor  of  a  National 
Academies  of  Sciences  study  on  improv¬ 
ing  processes  for  acquiring  and  testing 
IT  in  the  DoD.  We  should  have  their 
recommendations  in  November.  We’ve 
also  engaged  the  Business  Executives  for 
National  Security  (BENS)  and  the  National  Defense 
Industrial  Association  (NDIA)  to  hear  their  recom¬ 
mendations.  Recently,  a  Defense  Science  Board 
completed  a  study  on  IT  acquisition  and  made 
recommendations  to  institute  a  new  acquisition  model 
geared  specifically  to  IT.  One  common  theme 
emerging  from  these  studies  is  that  the  DoD  should 
adopt  agile  development  and  test  practices.  Nowhere  is 
the  “on-demand”  requirement  more  prevalent  than  in 
IT — ^whether  it’s  bandwidth,  spectrum,  storage  capac¬ 
ity,  intrusion  protection,  or  computing,  we  expect  our 
IT  capabilities  to  provide  what  we  need,  when  we  need 
them.  To  support  the  warfighter  with  relevant  IT 
today  requires  us  to  shift  how  we  do  business  to 
support  test  on  demand. 

DISA  is  an  acquisition  organization  for  Joint 
capabilities,  likewise  T8cE  of  Joint  capabilities  is  an 
essential  part  of  our  mission  in  DISA;  hence,  our 
second  goal.  Our  test  organizations  provide  full 
spectrum  T5cE  services  (developmental,  operational, 
interoperability,  and  security  T8cE)  across  all  of  the 
Joint  capability  areas.  It  is  imperative  that  we  get  this 
right.  The  SPG  speaks  to  “developing  and  testing  joint 
capability  portfolios,”  and  I  would  like  to  emphasize 
joinf  in  that  statement  because,  simply  put,  the 
processes  we  have  in  place  today  are  not  designed  to 
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support  Joint  programs.  To  accomplish  the  guidance, 
we  are  going  to  need  to  formalize  processes  for 
obtaining  Joint  troops  as  test  units,  approved  Joint 
mission  threads,  and  relevant  Joint  scenarios.  We  are 
working  with  Joint  Forces  Command  to  improve  these 
processes  for  Joint  testers. 

In  the  world  of  IT,  agility  is  the  name  of  the  game. 
It  frustrates  me  when  we  have  capability  improvements 
ready  to  go  but  can’t  get  them  fielded.  I  believe  this  has 
led  our  warfighting  units  to  circumvent  the  acquisition 
process  and  buy  commercial  capabilities.  Think  about 
that,  because  it  is  its  own  form  of  acquisition  process: 
the  commanders  in  the  field  weigh  the  risks  and  make 
a  decision — speed  is  important,  costs  low,  risks 
manageable,  capabilities  improve.  Our  DoD  acquisi¬ 
tion  process  should  do  the  same  thing,  and  our  T5cE 
organizations  will  have  to  be  better,  faster,  and  more 
mission-focused  than  ever  before.  It  is  essential  that  we 
combine  efforts  and  work  as  a  team  to  streamline  test 
processes  and  eliminate  redundancy.  Our  third  goal  is 
to  team  for  maximum  efficiency,  and  empower  the 
capability  test  teams  to  design,  execute,  and  report 
results  of  T5cE  activities. 

The  commercial  sector  succeeds  with  agile  methods 
due  to  teamwork  and  empowerment — the  opposite  of 
our  DoD  acquisition  process  today!  Fortunately,  our 
leaders  have  recognized  the  value  of  teamwork  in 
achieving  greater  efficiency.  Almost  2  years  ago,  in 
December  2007,  the  Director,  Operational  Test  and 
Evaluation,  the  Honorable  Dr.  Charles  McQueary, 
and  then  Undersecretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics,  the  Honorable  John  J. 
Young,  Jr.,  signed  into  policy  that  T5cE  “shall  be 
integrated  and  seamless  throughout  the  system  life- 
cycle.”  Integrated  testing,  the  theme  of  this  issue  of  The 
ITEA  Journal^  is  all  about  teamwork. 

In  the  IT  world,  we  have  a  lot  of  decision  makers  to 
satisfy  to  get  a  capability  fielded;  therefore,  our  test 
teams  should  be  composed  of  members  representing  all 
of  their  interests.  There’s  a  milestone  decision 
authority,  an  interoperability  certifier,  and  a  designated 
approving  authority  who  certifies  information  assur¬ 
ance.  A  typical  integrated  master  schedule  will  have 
developmental  test  events,  a  security  test  event,  and  an 
operational  test  event,  all  conducted  by  different 
organizations  and  spanning  a  period  of  several  months. 
An  integrated  test  model  for  IT  could  bring  all  test 
activities  together  and  cut  that  time  in  half  or  more.  I 
think  this  will  yield  considerable  cost  savings,  but  more 
important,  if  testers  work  as  a  team,  we  can  produce  a 
more  complete  evaluation  to  give  the  decision  makers. 

Integrated  testing  is  essential  to  the  transformation 
of  T5cE  to  the  “on  demand”  world.  For  programs 
following  agile  processes,  T5cE  has  to  be  on  demand. 


It  means  being  able  to  assemble  the  right  team  of 
skilled  testers,  translate  user  stories  into  an  effective 
test  design,  use  test  automation  as  much  as  possible, 
analyze  the  data,  and  share  the  results — all  in  a  period 
of  days,  not  weeks  or  months.  It  is  essential  that  our 
testers  understand  the  Joint  environment  and  be  able  to 
assemble  the  right  components  quickly  and  at  low  cost. 
To  test  capabilities  such  as  those  DISA  is  developing, 
we  need  an  on-demand  Joint  test  environment  with  the 
means  to  find  the  right  set  of  test  assets,  schedule  their 
use,  collect  data,  and  then  release  the  asset  when 
completed.  Those  who  oversee  these  agile  programs 
have  to  adjust  their  processes  accordingly,  which  means 
that  governance  bodies  such  as  the  overarching 
integrated  process  teams  (OIPTs)  have  to  be  able  to 
work  within  the  agile  framework. 

Last,  our  fourth  goal  is  the  development  and 
retention  of  the  workforce.  The  SPG  highlights 
training  as  part  of  the  evolution  toward  demand- 
driven  processes.  We  have  a  professional  acquisition 
workforce  and  a  quality  acquisition  university  to 
support  them.  However,  IT  acquisition  is  different; 
we  should  not  expect  our  program  managers  and  testers 
to  be  proficient  in  IT  acquisition  and  T5cE  when  we 
train  them  through  a  program  of  instruction  based  for 
the  most  part  on  weapons  platforms.  As  we  evolve 
training,  let’s  address  the  education  of  our  IT 
acquisition  and  T5cE  workforce.  This  is  one  of  many 
initiatives  DISA  is  pursuing. 

The  challenges  for  the  T8cE  community  are  great 
but  not  insurmountable.  Our  job  is  to  set  the 
conditions  for  success.  At  DISA  we’re  taking  on  all 
of  these  challenges,  but  we  know  that  we  don’t  have  all 
of  the  answers.  We  need  your  help  to  get  the  good 
ideas  turned  into  action.  Let’s  work  together  to  develop 
the  processes,  methods,  training,  and  infrastructure  to 
ensure  we  “test  like  we  fight.”  We  look  forward  to 
working  with  all  of  you  to  transform  acquisition  and 
testing  to  the  on-demand  world.  □ 

Steven  J.  Hutchison,  Ph.D.,  a  member  of  the  Senior 
Executive  Service,  is  the  Test  and  Evaluation  Executive 
for  the  Defense  Information  Systems  Agency.  He  is 
responsible  for  developing  and  enforcing  T^E  policy  and 
procedures,  representing  DISA  to  the  DoD  T^E 
community,  and  providing  direct  support  to  DISA 
programs  for  T&E  of  IT  capabilities.  Prior  to  his  arrival 
at  DISA,  Dr.  Hutchison  served  in  the  office  of  the 
Director,  Operational  Test  and  Evaluation  (DOT^E), 
Office  of  the  Secretary  of  Defense,  as  a  net-centric  warfare 
systems  analyst.  While  in  DOT&E,  he  had  oversight 
responsibilities  for  several  of  the  major  warfighting 
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information  systems,  including  the  Global  Command  and 
Control  System-Joint,  the  Service  variants  of  the 
Distributed  Common  Ground/Surface  System,  and  the 
Net  Enabled  Command  Capability.  Dr.  Hutchison  also 


served  for  6  years  in  the  Army  Test  and  Evaluation 
Command  as  an  evaluation  officer  and  later  as  the 
Assistant  Technical  Director.  E-mail:  Steven.Hutchison@ 
disa.mil 
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